{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5Otv1wvu-Ajp"
      },
      "source": [
        "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/learn/generation/langchain/handbook/05-langchain-retrieval-augmentation.ipynb)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RtCrQ_9T-Ajq"
      },
      "source": [
        "#### [LangChain Handbook](https://www.pinecone.io/learn/series/langchain/)\n",
        "\n",
        "# Retrieval Augmentation\n",
        "\n",
        "**L**arge **L**anguage **M**odels (LLMs) have a data freshness problem. Even the most powerful LLMs in the world are not up to date with recent world events.\n",
        "\n",
        "The world of LLMs is frozen in time. Their world exists as a static snapshot of the world as it was within their training data.\n",
        "\n",
        "A solution to this problem is *retrieval augmentation*. The idea behind this is that we retrieve relevant information from an external knowledge base and give that information to our LLM. In this notebook we will learn how to do that.\n",
        "\n",
        "[![Open fast notebook](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/fast-link.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/generation/langchain/handbook/05-langchain-retrieval-augmentation-fast.ipynb)\n",
        "\n",
        "To begin, we must install the prerequisite libraries that we will be using in this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 22,
      "metadata": {
        "id": "1EL87M7Z-Ajq"
      },
      "outputs": [],
      "source": [
        "!pip install -qU \\\n",
        "    datasets==3.6.0 \\\n",
        "    langchain==0.3.25 \\\n",
        "    langchain-openai==0.3.22 \\\n",
        "    langchain-pinecone==0.2.8 \\\n",
        "    tiktoken==0.9.0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WSH274YG-Ajq"
      },
      "source": [
        "## Building the Knowledge Base"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DiRWzKh0mMGv",
        "outputId": "0c258537-1dc5-4c09-c9f8-bdc7ae062139"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "/usr/local/lib/python3.11/dist-packages/huggingface_hub/utils/_auth.py:94: UserWarning: \n",
            "The secret `HF_TOKEN` does not exist in your Colab secrets.\n",
            "To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n",
            "You will be able to reuse this secret in all of your notebooks.\n",
            "Please note that authentication is recommended but still optional to access public models or datasets.\n",
            "  warnings.warn(\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "Dataset({\n",
              "    features: ['id', 'subreddit', 'title', 'selftext'],\n",
              "    num_rows: 107\n",
              "})"
            ]
          },
          "metadata": {},
          "execution_count": 2
        }
      ],
      "source": [
        "from datasets import load_dataset\n",
        "\n",
        "data = load_dataset(\n",
        "    \"aurelio-ai/reddit-finance\",\n",
        "    split=\"train\",\n",
        ")\n",
        "data"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "LarkabZgtbhQ",
        "outputId": "e7e92276-4daf-4c1a-a73a-21a4145e6b58"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'id': '1k7782l',\n",
              " 'subreddit': 'stocks',\n",
              " 'title': 'Waymo reports 250,000 paid robotaxi rides per week in U.S.',\n",
              " 'selftext': 'Alphabet\\xa0reported Thursday that Waymo, its autonomous vehicle unit, is now delivering more than 250,000 paid robotaxi rides per week in the U.S.\\n\\nCEO Sundar Pichai said Waymo has options in terms of “business models across geographies,” and the robotaxi company is building partnerships with ride-hailing app Uber, automakers and operations and maintenance businesses that tend to its vehicle fleets.\\n\\n“We can’t possibly do it all ourselves,” said Pichai on a call with analysts for Alphabet’s\\xa0first-quarter earnings.\\xa0\\n\\nPichai noted that Waymo has not entirely defined its long-term business model, and there is “future optionality around personal ownership” of vehicles equipped with Waymo’s self-driving technology. The company is also exploring the ways it can scale up its operations, he said.\\n\\nThe 250,000 paid rides per week are up from 200,000 in February, before Waymo opened in\\xa0Austin\\xa0and expanded in the\\xa0San Francisco Bay Area\\xa0in March.\\xa0\\n\\nWaymo, which is part of Alphabet’s Other Bets segment, is already running its commercial, driverless ride-hailing services in the San Francisco, Los Angeles, Phoenix and Austin regions.\\n\\nEarlier this month, Waymo and its partner Uber, began allowing interested riders to sign up to try the robotaxi service in Atlanta when it opens this summer.\\xa0\\n\\nThe early pioneer in self-driving technology, Waymo has managed to beat Elon Musk-led\\xa0Tesla\\xa0and a myriad of now-defunct autonomous vehicle startups to the U.S. market.\\n\\nTesla is promising that it will be able to turn its Model Y SUVs into robotaxis by the end of June for a driverless ride-hailing service it plans to launch in Austin.\\n\\nAfter about a decade of promises and missed deadlines, Tesla\\xa0still does not offer a vehicle that’s safe to use without a human at the wheel ready to steer or brake at all times.\\n\\nMusk criticized Waymo’s approach to driverless tech on his company’s\\xa0first-quarter earnings\\xa0call on Tuesday. Musk said Waymo autonomous vehicles are “very expensive” and made in only “low volume.” Tesla’s partially automated driving systems rely mostly on cameras to navigate, while Waymo’s driverless systems rely on lidar technology, other sensors and cameras.\\n\\nWould-be competitors to Waymo also include\\xa0Amazon-owned Zoox, Mobileye, May Mobility and international autonomous vehicle companies such as WeRide and Baidu’s Apollo Go.\\n\\nSource: [https://www.cnbc.com/2025/04/24/waymo-reports-250000-paid-robotaxi-rides-per-week-in-us.html](https://www.cnbc.com/2025/04/24/waymo-reports-250000-paid-robotaxi-rides-per-week-in-us.html)'}"
            ]
          },
          "metadata": {},
          "execution_count": 3
        }
      ],
      "source": [
        "data[5]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OPpcO-TwuQwD"
      },
      "source": [
        "Many records contain *a lot* of text. Our first task is therefore to identify a good preprocessing methodology for chunking these articles into more \"concise\" chunks to later be embedding and stored in our Pinecone vector database.\n",
        "\n",
        "For this we use LangChain's `RecursiveCharacterTextSplitter` to split our text into chunks of a specified max length."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "id": "PWC58VQr-Ajq"
      },
      "outputs": [],
      "source": [
        "import tiktoken\n",
        "\n",
        "# We use gpt-4.1-mini as standard but tiktoken does not support gpt-4.1.\n",
        "# Fortunately, 4.1 and 4o models all use the same underlying tokenizer and so\n",
        "# we can use gpt-4o here\n",
        "encoding = tiktoken.encoding_for_model('gpt-4o')"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qIbWhJ1j-Ajq"
      },
      "source": [
        "The tokenizer encoding that we'll use is:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 36
        },
        "id": "P9k7zr-m-Ajq",
        "outputId": "3d1aaa26-a0f7-41df-84ae-fe0f962a5bea"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'o200k_base'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 5
        }
      ],
      "source": [
        "encoding.name"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "a3ChSxlcwX8n",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "43c6b189-4d16-4988-b806-52d66bd3f5dc"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "27"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ],
      "source": [
        "import tiktoken\n",
        "\n",
        "tokenizer = tiktoken.get_encoding(encoding.name)\n",
        "\n",
        "# create the length function\n",
        "def tiktoken_len(text):\n",
        "    tokens = tokenizer.encode(\n",
        "        text,\n",
        "        disallowed_special=()\n",
        "    )\n",
        "    return len(tokens)\n",
        "\n",
        "tiktoken_len(\"hello I am a chunk of text and using the tiktoken_len function \"\n",
        "             \"we can find the length of this chunk of text in tokens\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 7,
      "metadata": {
        "id": "58J-y6GHtvQP"
      },
      "outputs": [],
      "source": [
        "from langchain.text_splitter import RecursiveCharacterTextSplitter\n",
        "\n",
        "text_splitter = RecursiveCharacterTextSplitter(\n",
        "    chunk_size=300,\n",
        "    chunk_overlap=20,\n",
        "    length_function=tiktoken_len,\n",
        "    separators=[\"\\n\\n\", \"\\n\", \" \", \"\"]\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "W8KGqv-rzEgH",
        "outputId": "290abf80-f512-4037-c2a4-a2187fb99691"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "['Alphabet\\xa0reported Thursday that Waymo, its autonomous vehicle unit, is now delivering more than 250,000 paid robotaxi rides per week in the U.S.\\n\\nCEO Sundar Pichai said Waymo has options in terms of “business models across geographies,” and the robotaxi company is building partnerships with ride-hailing app Uber, automakers and operations and maintenance businesses that tend to its vehicle fleets.\\n\\n“We can’t possibly do it all ourselves,” said Pichai on a call with analysts for Alphabet’s\\xa0first-quarter earnings.\\xa0\\n\\nPichai noted that Waymo has not entirely defined its long-term business model, and there is “future optionality around personal ownership” of vehicles equipped with Waymo’s self-driving technology. The company is also exploring the ways it can scale up its operations, he said.\\n\\nThe 250,000 paid rides per week are up from 200,000 in February, before Waymo opened in\\xa0Austin\\xa0and expanded in the\\xa0San Francisco Bay Area\\xa0in March.\\xa0\\n\\nWaymo, which is part of Alphabet’s Other Bets segment, is already running its commercial, driverless ride-hailing services in the San Francisco, Los Angeles, Phoenix and Austin regions.\\n\\nEarlier this month, Waymo and its partner Uber, began allowing interested riders to sign up to try the robotaxi service in Atlanta when it opens this summer.',\n",
              " 'The early pioneer in self-driving technology, Waymo has managed to beat Elon Musk-led\\xa0Tesla\\xa0and a myriad of now-defunct autonomous vehicle startups to the U.S. market.\\n\\nTesla is promising that it will be able to turn its Model Y SUVs into robotaxis by the end of June for a driverless ride-hailing service it plans to launch in Austin.\\n\\nAfter about a decade of promises and missed deadlines, Tesla\\xa0still does not offer a vehicle that’s safe to use without a human at the wheel ready to steer or brake at all times.\\n\\nMusk criticized Waymo’s approach to driverless tech on his company’s\\xa0first-quarter earnings\\xa0call on Tuesday. Musk said Waymo autonomous vehicles are “very expensive” and made in only “low volume.” Tesla’s partially automated driving systems rely mostly on cameras to navigate, while Waymo’s driverless systems rely on lidar technology, other sensors and cameras.\\n\\nWould-be competitors to Waymo also include\\xa0Amazon-owned Zoox, Mobileye, May Mobility and international autonomous vehicle companies such as WeRide and Baidu’s Apollo Go.\\n\\nSource: [https://www.cnbc.com/2025/04/24/waymo-reports-250000-paid-robotaxi-rides-per-week-in-us.html](https://www.cnbc.com/2025/04/24/waymo-reports-250000-paid-robotaxi-rides-per-week-in-us.html)']"
            ]
          },
          "metadata": {},
          "execution_count": 8
        }
      ],
      "source": [
        "chunks = text_splitter.split_text(data[5]['selftext'])\n",
        "chunks"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "K9hdjy22zVuJ",
        "outputId": "7d32539f-2eb2-4829-dcc2-ab4f0757d932"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(279, 291)"
            ]
          },
          "metadata": {},
          "execution_count": 9
        }
      ],
      "source": [
        "tiktoken_len(chunks[0]), tiktoken_len(chunks[1])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SvApQNma0K8u"
      },
      "source": [
        "Using the `text_splitter` we get much better sized chunks of text. We'll use this functionality during the indexing process later. Now let's take a look at embedding.\n",
        "\n",
        "## Creating Embeddings\n",
        "\n",
        "Building embeddings using LangChain's OpenAI embedding support is fairly straightforward. We first need to add our [OpenAI api key]() by running the next cell:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dphi6CC33p62",
        "outputId": "91dd14b5-5b6a-4687-f2b2-9bf80aa3edf8"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your OpenAI API key: ··········\n"
          ]
        }
      ],
      "source": [
        "import os\n",
        "from getpass import getpass\n",
        "\n",
        "os.environ[\"OPENAI_API_KEY\"] = os.getenv(\"OPENAI_API_KEY\") \\\n",
        "    or getpass(\"Enter your OpenAI API key: \")\n",
        "\n",
        "OPENAI_API_KEY = os.getenv(\"OPENAI_API_KEY\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "49hoj_ZS3wAr"
      },
      "source": [
        "*(Note that OpenAI is a paid service and so running the remainder of this notebook may incur some small cost)*\n",
        "\n",
        "After initializing the API key we can initialize our `text-embedding-3-small` embedding model like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 11,
      "metadata": {
        "id": "mBLIWLkLzyGi"
      },
      "outputs": [],
      "source": [
        "from langchain_openai import OpenAIEmbeddings\n",
        "\n",
        "model_name = 'text-embedding-3-small'\n",
        "\n",
        "embed = OpenAIEmbeddings(\n",
        "    model=model_name,\n",
        "    openai_api_key=OPENAI_API_KEY\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SwbZGT-v4iMi"
      },
      "source": [
        "Now we embed some text like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 12,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vM-HuKtl4cyt",
        "outputId": "420e3bc0-ca37-431e-ba89-83c5170df901"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(2, 1536)"
            ]
          },
          "metadata": {},
          "execution_count": 12
        }
      ],
      "source": [
        "texts = [\n",
        "    'this is the first chunk of text',\n",
        "    'then another second chunk of text is here'\n",
        "]\n",
        "\n",
        "res = embed.embed_documents(texts)\n",
        "len(res), len(res[0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QPUmWYSA43eC"
      },
      "source": [
        "From this we get *two* (aligning to our two chunks of text) 1536-dimensional embeddings.\n",
        "\n",
        "Now we move on to initializing our Pinecone vector database.\n",
        "\n",
        "## Vector Database\n",
        "\n",
        "To create our vector database we first need a [free API key from Pinecone](https://app.pinecone.io). Then we initialize like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 13,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QN8ys_qW-Ajr",
        "outputId": "4b71b734-c085-462f-88b5-f517fb59b859"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Enter your Pinecone API key: ··········\n"
          ]
        }
      ],
      "source": [
        "from pinecone import Pinecone\n",
        "import os\n",
        "\n",
        "\n",
        "import os\n",
        "from pinecone import Pinecone\n",
        "from getpass import getpass\n",
        "\n",
        "os.environ[\"PINECONE_API_KEY\"] = os.getenv(\"PINECONE_API_KEY\") \\\n",
        "    or getpass(\"Enter your Pinecone API key: \")\n",
        "\n",
        "PINECONE_API_KEY = os.getenv(\"PINECONE_API_KEY\")\n",
        "\n",
        "# configure client\n",
        "pc = Pinecone(api_key=PINECONE_API_KEY)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i0lddCIr-Ajr"
      },
      "source": [
        "Then we initialize the index. We will be using OpenAI's `text-embedding-3-small` model for creating the embeddings, so we set the `dimension` to `1536`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 24,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "9pT9C4nW4vwo",
        "outputId": "436f4e6e-0ca7-4b65-c8d6-ec3adfb8d6b0"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'dimension': 1536,\n",
              " 'index_fullness': 0.0,\n",
              " 'metric': 'dotproduct',\n",
              " 'namespaces': {},\n",
              " 'total_vector_count': 0,\n",
              " 'vector_type': 'dense'}"
            ]
          },
          "metadata": {},
          "execution_count": 24
        }
      ],
      "source": [
        "from pinecone import AwsRegion, CloudProvider, Metric, ServerlessSpec\n",
        "\n",
        "index_name = 'langchain-retrieval-augmentation'\n",
        "\n",
        "# check if index already exists (it shouldn't if this is first time)\n",
        "if not pc.has_index(name=index_name):\n",
        "    # if does not exist, create index\n",
        "    pc.create_index(\n",
        "        name=index_name,\n",
        "        dimension=1536,  # dimensionality of text-embedding-3-small\n",
        "        metric=Metric.DOTPRODUCT,\n",
        "        spec=ServerlessSpec(\n",
        "            cloud=CloudProvider.AWS,\n",
        "            region=AwsRegion.US_EAST_1\n",
        "        )\n",
        "    )\n",
        "\n",
        "# connect to index\n",
        "index = pc.Index(name=index_name)\n",
        "# view index stats\n",
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0RqIF2mIDwFu"
      },
      "source": [
        "We should see that the new Pinecone index has a `total_vector_count` of `0`, as we haven't added any vectors yet.\n",
        "\n",
        "## Indexing\n",
        "\n",
        "We can perform the indexing task using the LangChain vector store object or via the Pinecone python client directly. Here, we will do this via the Pinecone client, upserting our records in batches of `100` or more."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 25,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 49,
          "referenced_widgets": [
            "b5971a89450d4fb4ad180bccffee1b6a",
            "18e0bfa486c446b88bfd0d9178b6e3db",
            "61fbcbd5d4cc4dfbbf7b5cecaa61e65c",
            "9f2a47a250bf4d9cbe8a1754b9fd6dd0",
            "73b5423342644d8ea1e12223ba142389",
            "fdb025c270294f68b47c2a80cf9e8fa0",
            "2c99d5cf84004e2c81c465f2c793550f",
            "9cb0121d277f473997cf6675b4de2feb",
            "35d1efe7e519415a81923e014a494f8b",
            "88f1a9315dd2477cb8f1e0d301e84005",
            "933afb4378064ad49cd9c81de0368575"
          ]
        },
        "id": "W-cIOoTWGY1R",
        "outputId": "e5b29a31-caed-4a51-88a9-52fb66bb18c4"
      },
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "  0%|          | 0/107 [00:00<?, ?it/s]"
            ],
            "application/vnd.jupyter.widget-view+json": {
              "version_major": 2,
              "version_minor": 0,
              "model_id": "b5971a89450d4fb4ad180bccffee1b6a"
            }
          },
          "metadata": {}
        }
      ],
      "source": [
        "from tqdm.auto import tqdm\n",
        "from uuid import uuid4\n",
        "\n",
        "batch_limit = 100\n",
        "\n",
        "texts = []\n",
        "metadatas = []\n",
        "\n",
        "for i, record in enumerate(tqdm(data)):\n",
        "    # first get metadata fields for this record\n",
        "    url = f\"https://reddit.com/r/{record['subreddit']}/comments/{record['id']}\"\n",
        "    metadata = {\n",
        "        'thread_id': str(record['id']),\n",
        "        'source': url,\n",
        "        'subreddit': record['subreddit']\n",
        "    }\n",
        "    # now we create chunks from the record text\n",
        "    record_texts = text_splitter.split_text(record['selftext'])\n",
        "    # create individual metadata dicts for each chunk\n",
        "    record_metadatas = [{\n",
        "        \"chunk\": j, \"text\": text, **metadata\n",
        "    } for j, text in enumerate(record_texts)]\n",
        "    # append these to current batches\n",
        "    texts.extend(record_texts)\n",
        "    metadatas.extend(record_metadatas)\n",
        "    # if we have reached the batch_limit we can add texts\n",
        "    if len(texts) >= batch_limit:\n",
        "        ids = [str(uuid4()) for _ in range(len(texts))]\n",
        "        embeds = embed.embed_documents(texts)\n",
        "        index.upsert(vectors=zip(ids, embeds, metadatas))\n",
        "        texts = []\n",
        "        metadatas = []\n",
        "\n",
        "if len(texts) > 0:\n",
        "    ids = [str(uuid4()) for _ in range(len(texts))]\n",
        "    embeds = embed.embed_documents(texts)\n",
        "    index.upsert(vectors=zip(ids, embeds, metadatas))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XaF3daSxyCwB"
      },
      "source": [
        "We've now indexed everything. We can check the number of vectors in our index like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 29,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CaEBhsAM22M3",
        "outputId": "6a54bc5a-3d60-45bb-f5b0-0db6b062e208"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "{'dimension': 1536,\n",
              " 'index_fullness': 0.0,\n",
              " 'metric': 'dotproduct',\n",
              " 'namespaces': {'': {'vector_count': 170}},\n",
              " 'total_vector_count': 170,\n",
              " 'vector_type': 'dense'}"
            ]
          },
          "metadata": {},
          "execution_count": 29
        }
      ],
      "source": [
        "index.describe_index_stats()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-8P2PryCy8W3"
      },
      "source": [
        "## Creating a Vector Store and Querying\n",
        "\n",
        "Now that we've build our index we can switch back over to LangChain. We start by initializing a vector store using the same index we just built. We do that like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 30,
      "metadata": {
        "id": "qMXlvXOAyJHy"
      },
      "outputs": [],
      "source": [
        "from langchain_pinecone import PineconeVectorStore\n",
        "\n",
        "# initialize the vector store object\n",
        "vectorstore = PineconeVectorStore(index=index, embedding=embed)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 31,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "COT5s7hcyPiq",
        "outputId": "cbbb63de-b764-48a8-994b-4129bdcb81c9"
      },
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[Document(id='1f8c9fe9-cbf9-48e4-81fc-a1528faed668', metadata={'chunk': 0.0, 'source': 'https://reddit.com/r/stocks/comments/1k7782l', 'subreddit': 'stocks', 'thread_id': '1k7782l'}, page_content='Alphabet\\xa0reported Thursday that Waymo, its autonomous vehicle unit, is now delivering more than 250,000 paid robotaxi rides per week in the U.S.\\n\\nCEO Sundar Pichai said Waymo has options in terms of “business models across geographies,” and the robotaxi company is building partnerships with ride-hailing app Uber, automakers and operations and maintenance businesses that tend to its vehicle fleets.\\n\\n“We can’t possibly do it all ourselves,” said Pichai on a call with analysts for Alphabet’s\\xa0first-quarter earnings.\\xa0\\n\\nPichai noted that Waymo has not entirely defined its long-term business model, and there is “future optionality around personal ownership” of vehicles equipped with Waymo’s self-driving technology. The company is also exploring the ways it can scale up its operations, he said.\\n\\nThe 250,000 paid rides per week are up from 200,000 in February, before Waymo opened in\\xa0Austin\\xa0and expanded in the\\xa0San Francisco Bay Area\\xa0in March.\\xa0\\n\\nWaymo, which is part of Alphabet’s Other Bets segment, is already running its commercial, driverless ride-hailing services in the San Francisco, Los Angeles, Phoenix and Austin regions.\\n\\nEarlier this month, Waymo and its partner Uber, began allowing interested riders to sign up to try the robotaxi service in Atlanta when it opens this summer.'),\n",
              " Document(id='6d67bbcd-0e41-4665-8ea0-79a9957bdb38', metadata={'chunk': 1.0, 'source': 'https://reddit.com/r/stocks/comments/1k7782l', 'subreddit': 'stocks', 'thread_id': '1k7782l'}, page_content='The early pioneer in self-driving technology, Waymo has managed to beat Elon Musk-led\\xa0Tesla\\xa0and a myriad of now-defunct autonomous vehicle startups to the U.S. market.\\n\\nTesla is promising that it will be able to turn its Model Y SUVs into robotaxis by the end of June for a driverless ride-hailing service it plans to launch in Austin.\\n\\nAfter about a decade of promises and missed deadlines, Tesla\\xa0still does not offer a vehicle that’s safe to use without a human at the wheel ready to steer or brake at all times.\\n\\nMusk criticized Waymo’s approach to driverless tech on his company’s\\xa0first-quarter earnings\\xa0call on Tuesday. Musk said Waymo autonomous vehicles are “very expensive” and made in only “low volume.” Tesla’s partially automated driving systems rely mostly on cameras to navigate, while Waymo’s driverless systems rely on lidar technology, other sensors and cameras.\\n\\nWould-be competitors to Waymo also include\\xa0Amazon-owned Zoox, Mobileye, May Mobility and international autonomous vehicle companies such as WeRide and Baidu’s Apollo Go.\\n\\nSource: [https://www.cnbc.com/2025/04/24/waymo-reports-250000-paid-robotaxi-rides-per-week-in-us.html](https://www.cnbc.com/2025/04/24/waymo-reports-250000-paid-robotaxi-rides-per-week-in-us.html)'),\n",
              " Document(id='5bca9455-46b8-4ee0-9587-bc97cab119ee', metadata={'chunk': 0.0, 'source': 'https://reddit.com/r/pennystocks/comments/1k70v8k', 'subreddit': 'pennystocks', 'thread_id': '1k70v8k'}, page_content=\"This two-wheel electric vehicle company could make for a quick return. Currently, the stock price sits at $0.0035 (as I type this), but they have $100 million in pre-orders for their two motorcycle models, which include safety and accessibility tech you'd find in newer four-wheel vehicles. They're also big on data intelligence, and are open to working with other companies to share info, such as supporting ICE (internal combustion engine) companies to transition. They presented at the Planet MicroCap Showcase just yesterday, which is available online: [https://event.summitcast.com/view/YNz6mnmEsXyrdRxb78w2nX/9WosXgUD44wqvJunNd982Y](https://event.summitcast.com/view/YNz6mnmEsXyrdRxb78w2nX/9WosXgUD44wqvJunNd982Y) (Or just Google it if you don't trust links.) Among other things, they state the global market for electric bikes is forecasted to reach $40 billion by 2030. While the US may have an administration trying to move in the opposite direction of electrification, the world thinks otherwise!\")]"
            ]
          },
          "metadata": {},
          "execution_count": 31
        }
      ],
      "source": [
        "query = \"how many robotaxi rides did waymo report in the US?\"\n",
        "\n",
        "vectorstore.similarity_search(\n",
        "    query,  # our search query\n",
        "    k=3  # return 3 most relevant docs\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZCvtmREd0pdo"
      },
      "source": [
        "All of these are good, relevant results. But what can we do with this? There are many tasks, one of the most interesting (and well supported by LangChain) is called _\"Retrieval Augmented Generation\"_ or RAG.\n",
        "\n",
        "## Retrieval Augmented Generation\n",
        "\n",
        "In RAG we take the query as a question that is to be answered by a LLM, but the LLM must answer the question based on the information it is seeing being returned from the `vectorstore`.\n",
        "\n",
        "To do this we initialize a retrieval pipeline like so:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 32,
      "metadata": {
        "id": "moCvQR-p0Zsb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 53
        },
        "outputId": "6189343e-f28b-451c-d3d0-05ce58434ecf"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "how many robotaxi rides did waymo report in the US?\n"
          ]
        },
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "'Waymo reported delivering more than 250,000 paid robotaxi rides per week in the U.S.'"
            ],
            "application/vnd.google.colaboratory.intrinsic+json": {
              "type": "string"
            }
          },
          "metadata": {},
          "execution_count": 32
        }
      ],
      "source": [
        "from langchain_openai import ChatOpenAI\n",
        "from langchain_core.prompts import ChatPromptTemplate\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "\n",
        "# completion llm\n",
        "llm = ChatOpenAI(\n",
        "    openai_api_key=OPENAI_API_KEY,\n",
        "    model_name='gpt-4.1-mini',\n",
        "    temperature=0.0\n",
        ")\n",
        "\n",
        "# Create prompt template\n",
        "template = \"\"\"Answer the question based on the following context:\n",
        "\n",
        "{context}\n",
        "\n",
        "Question: {question}\n",
        "\n",
        "Answer:\"\"\"\n",
        "\n",
        "prompt = ChatPromptTemplate.from_template(template)\n",
        "\n",
        "# Create LCEL chain\n",
        "retrieval_chain = (\n",
        "    {\"context\": vectorstore.as_retriever(), \"question\": lambda x: x}\n",
        "    | prompt\n",
        "    | llm\n",
        "    | StrOutputParser()\n",
        ")\n",
        "\n",
        "print(query)\n",
        "\n",
        "retrieval_chain.invoke(query)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0qf5e3xf3ggq"
      },
      "source": [
        "We can also include the sources of information that the LLM is using to answer our question:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 35,
      "metadata": {
        "id": "aYVMGDA13cTz",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "6906ed41-3bbb-46de-911c-37311fb45b3f"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Question: how many robotaxi rides did waymo report in the US?\n",
            "\n",
            "Waymo reported delivering more than 250,000 paid robotaxi rides per week in the U.S. This figure is an increase from 200,000 rides per week in February, following Waymo's expansion into Austin and the San Francisco Bay Area in March.\n",
            "\n",
            "Sources:  \n",
            "- https://reddit.com/r/stocks/comments/1k7782l  \n",
            "- https://www.cnbc.com/2025/04/24/waymo-reports-250000-paid-robotaxi-rides-per-week-in-us.html\n"
          ]
        }
      ],
      "source": [
        "from langchain_core.prompts import ChatPromptTemplate\n",
        "from langchain_core.output_parsers import StrOutputParser\n",
        "\n",
        "# Create prompt template with source formatting\n",
        "template = \"\"\"Answer the question based on the following context. Include the source URLs in your answer.\n",
        "\n",
        "{context}\n",
        "\n",
        "Question: {question}\n",
        "\n",
        "Answer:\"\"\"\n",
        "\n",
        "prompt = ChatPromptTemplate.from_template(template)\n",
        "\n",
        "# Create LCEL chain with source formatting\n",
        "def format_docs(docs):\n",
        "    return \"\\\\n\\\\n\".join([f\"Source: {doc.metadata.get('source', 'Unknown')}\\\\n{doc.page_content}\" for doc in docs])\n",
        "\n",
        "retrieval_chain_with_sources = (\n",
        "    {\"context\": vectorstore.as_retriever() | format_docs, \"question\": lambda x: x}\n",
        "    | prompt\n",
        "    | llm\n",
        "    | StrOutputParser()\n",
        ")\n",
        "\n",
        "print(f\"Question: {query}\\n\")\n",
        "print(retrieval_chain_with_sources.invoke(query))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nMRk_P3Q7l5J"
      },
      "source": [
        "Now we answer the question being asked, *and* return the source of this information being used by the LLM."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "taky9do--Ajw"
      },
      "source": [
        "Delete the index to save resources when you're done!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 36,
      "metadata": {
        "id": "98w49Pdd-Ajw"
      },
      "outputs": [],
      "source": [
        "pc.delete_index(index_name)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ehJEn68qADoH"
      },
      "source": [
        "---"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "pinecone1",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.4"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "b5971a89450d4fb4ad180bccffee1b6a": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HBoxModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_18e0bfa486c446b88bfd0d9178b6e3db",
              "IPY_MODEL_61fbcbd5d4cc4dfbbf7b5cecaa61e65c",
              "IPY_MODEL_9f2a47a250bf4d9cbe8a1754b9fd6dd0"
            ],
            "layout": "IPY_MODEL_73b5423342644d8ea1e12223ba142389"
          }
        },
        "18e0bfa486c446b88bfd0d9178b6e3db": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_fdb025c270294f68b47c2a80cf9e8fa0",
            "placeholder": "​",
            "style": "IPY_MODEL_2c99d5cf84004e2c81c465f2c793550f",
            "value": "100%"
          }
        },
        "61fbcbd5d4cc4dfbbf7b5cecaa61e65c": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "FloatProgressModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_9cb0121d277f473997cf6675b4de2feb",
            "max": 107,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_35d1efe7e519415a81923e014a494f8b",
            "value": 107
          }
        },
        "9f2a47a250bf4d9cbe8a1754b9fd6dd0": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "HTMLModel",
          "model_module_version": "1.5.0",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_88f1a9315dd2477cb8f1e0d301e84005",
            "placeholder": "​",
            "style": "IPY_MODEL_933afb4378064ad49cd9c81de0368575",
            "value": " 107/107 [00:02&lt;00:00, 50.81it/s]"
          }
        },
        "73b5423342644d8ea1e12223ba142389": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "fdb025c270294f68b47c2a80cf9e8fa0": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "2c99d5cf84004e2c81c465f2c793550f": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "9cb0121d277f473997cf6675b4de2feb": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "35d1efe7e519415a81923e014a494f8b": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "ProgressStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "88f1a9315dd2477cb8f1e0d301e84005": {
          "model_module": "@jupyter-widgets/base",
          "model_name": "LayoutModel",
          "model_module_version": "1.2.0",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "933afb4378064ad49cd9c81de0368575": {
          "model_module": "@jupyter-widgets/controls",
          "model_name": "DescriptionStyleModel",
          "model_module_version": "1.5.0",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}